Operation control device and operation control method

ABSTRACT

There is provided an operation control device including a motion detection part which detects an object to be detected, which is moved by motion of a user, a motion determination part which determines motion of the object to be detected based on a detection result, a movable region movement processing part which moves a cursor movable region including a cursor operating an object displayed in a display region, and a cursor movement processing part which moves the cursor. Based on motion of a first detected object, the movable region movement processing part moves the cursor movable region along with the cursor in the display region by a first movement unit. Based on motion of a second detected object, the cursor movement processing part moves only the cursor in the cursor movable region by a second movement unit smaller than the first movement unit.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to an operation control device and anoperation control method.

2. Description of the Related Art

In the related art, when equipment is remotely operated, there has beenused a remote controller using infrared rays or a radio frequency (RF)signal or a pointing device such as a mouse, a joystick, and atrackball. For example, in the operation of a television, a remotecontroller is generally used, and, for example, the channel, the soundvolume, the hue, the density, and the contrast of the television can beoperated. In light of many objects to be controlled, the futurediversification of the broadcast service, and a conversion of atelevision to PC, the operation of the television is expected to be morecomplicated. Consequently, it is considered that convenience for usersis reduced, such as the number of operation keys of a remote controlleris increased, the functions are complicated, and the remote controlleris increased in size.

In order to solve the above issue, an input device as operation means ofequipment has been recently disclosed (for example, Japanese PatentApplication Laid-Open No. 5-324181). A user remotely operates the inputdevice by his fingers to directly input information to the input devicewithout using a device such as a remote controller. For example, asshown in FIG. 10, when an input device 10 is used, a user P moves afinger F in front of a display screen 28 of a television receiver 26,whereby information can be input to the input device 10. The inputdevice 10 detects the position of the finger F by means of a spatialposition detector 22 such as a stereo camera. A control part 24 thendisplays a cursor on a two-dimensional coordinate of the display screencorresponding to the position detected by the spatial position detector22. Moreover, the control part 24 moves the cursor in response to themovement of the finger F. When the user P moves the finger F to therebymove the cursor, and, thus, to select an execution button displayed onthe display screen 28, the operation related to the execution button isperformed by the control part 24.

SUMMARY OF THE INVENTION

However, in the related art, since the result output from the spatialposition detector detecting the finger position is used as it is as thecursor position on the display screen, the motion of the cursor on thedisplay screen may not correspond to the motion of the user's finger.For example, due to motion of fingers that is not intended by a user,such as a hand tremor, the action of a background, and a false detectionby the spatial position detector in a space with low detection accuracy,the motion of the cursor on the display screen may not correspond to themotion of the user's finger. In that case, the cursor on the displayscreen deviates from an orbit intended by the user, and thus it isdifficult for the user to accurately point the target execution button.

In gesture recognition, a cursor trace as a basis for the recognition isconstituted of continuous coordinate data groups. Thus, such an issuethat the motion of the cursor does not correspond to the motion ofuser's fingers is more serious in gesture functions.

The issue is more noticeable as a fineness of the resolution of atelevision is higher. Thus, while such a fineness that a cursor isoperated by a mouse is maintained, an interface including intuitivenessby a spatial gesture is required.

In light of the foregoing, it is desirable to provide a novel andimproved operation control device and operation control method, whichcan realize the intuitive and highly accurate operation of anoperational object.

According to an embodiment of the present invention, there is providedan operation control device including a motion detection part whichdetects an object to be detected, which is moved by motion of a user, amotion determination part which determines motion of the object to bedetected based on a detection result obtained by the motion detectionpart, a movable region movement processing part which moves a cursormovable region including a cursor operating an object displayed in adisplay region, and a cursor movement processing part which moves thecursor. The operation control device based on the motion of a firstdetected object detected by the motion detection part, the movableregion movement processing part moves the cursor movable region alongwith the cursor in the display region by a first movement unit, based onthe motion of a second detected object detected by the motion detectionpart, the cursor movement processing part moves only the cursor in thecursor movable region by a second movement unit smaller than the firstmovement unit.

According to the present invention, based on the motion of the firstdetected object, the cursor movable region and the cursor are moved bythe first movement unit, and based on the motion of the second detectedobject, the cursor can be moved by the second movement unit smaller thanthe first movement unit. The stepwise movement of the cursor can realizethe intuitively rough movement of the cursor and the fine movement ofthe cursor.

Here, when a region formed by dividing the display region into apredetermined number of regions is a first unit region, the cursormovable region includes one or two or more of the first unit regions,and the movable region movement processing part may move the cursormovable region along with the cursor in the unit of the first unitregion in accordance with an amount of movement of the first detectedobject.

Moreover, when a region formed by dividing the cursor movable regioninto a predetermined number of regions is a second unit region, thecursor movement processing part may move the cursor in the unit of thesecond unit region in accordance with an amount of movement of thesecond detected object.

Furthermore, when the cursor is located outside the cursor movableregion when the cursor is moved by the second detected object, themovable region movement processing part may move the cursor movableregion along with the cursor moved by the second detected object.

Moreover, only when the first detected object is in a movement mode thatenables movement of the cursor movable region, the movable regionmovement processing part may move the cursor movable region, and onlywhen the second detected object is in a movement mode that enablesmovement of the cursor, the cursor movement processing part may move thecursor.

Furthermore, the size of the cursor movable region moved by theoperation control device may be varied.

According to another embodiment of the present invention, there isprovided an operation control method, including the steps of detectingan object to be detected, which is moved by motion of a user,determining motion of the object to be detected based on a detectionresult, moving a cursor movable region, in which a cursor operating anobject displayed in a display region is displayed, and the cursortogether in the entire display region by a first movement unit based onmotion of a first detected object, and moving the cursor in the cursormovable region by a second movement unit smaller than the first movementunit based on motion of a second detected object.

As described above, the present invention can provide an operationcontrol device and an operation control method, which can realize theintuitive and highly accurate operation of an operational object.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is an explanatory view showing an example of a constitution of ascreen operated by a user's gesture according to an embodiment of thepresent invention;

FIG. 2 is a block diagram showing a hardware configuration of anoperation control device according to the embodiment;

FIG. 3 is a functional block diagram showing a functional configurationof the operation control device according to the present embodiment;

FIG. 4 is a flow chart showing an operation control method using theoperation control device according to the embodiment when right and lefthands are used;

FIG. 5 is an explanatory view explaining an operation of moving a cursormovable region;

FIG. 6 is an explanatory view explaining an operation of moving acursor;

FIG. 7 is a flow chart showing the operation control method using theoperation control device according to the embodiment when differentjoints are used;

FIG. 8 is a flow chart showing the operation control method using theoperation control device according to the embodiment when both hands andone hand are used;

FIG. 9 is a flow chart showing the operation control method using theoperation control device according to the embodiment when a user's sightline and his one hand are used; and

FIG. 10 is an explanatory view showing the related art input deviceusing a gesture.

DETAILED DESCRIPTION OF THE EMBODIMENT

Hereinafter, preferred embodiments of the present invention will bedescribed in detail with reference to the appended drawings. Note that,in this specification and the appended drawings, structural elementsthat have substantially the same function and structure are denoted withthe same reference numerals, and repeated explanation of thesestructural elements is omitted.

In addition, the description will be made in the following order.

1. Example of operating screen operated and controlled by operationcontrol device

2. Constitution of operation control device

3. Operation control method using operation control device

<1. Example of Operating Screen Operated and Controlled by OperationControl Device>

First, an example of operating a screen is described based on FIG. 1. Agesture of a user is recognized by an operation control device accordingto an embodiment of the present invention, and then the screen isoperated and controlled. FIG. 1 is an explanatory view showing anexample of a constitution of the screen operated by the user's gesture.

In the present embodiment, a target is selected by a free cursor used ina general personal computer and a mouse, and determination, cancel, anddisplay of menus are performed. For example, as shown in FIG. 1, asetting menu of a personal computer is displayed in a display region 200of the screen. A category view 202 showing a classification of settinginformation of the personal computer is displayed on the left side ofthe display region 200. When one category icon is selected from thecategory view 202, a view (a setting information view) 204 of thesetting information included in the selected category is displayed. Forexample, when a “display” category is selected from the category view202, the setting information view 204 including setting of a logo lamp,the screen, a color mode, and a screen saver included in the “display”category is displayed.

One setting information is selected from the setting information view204, and then when an “OK” button is selected from operation buttons206, the screen is changed to the setting screen of the selected settinginformation. Alternatively, when a “cancel” button is selected from theoperation buttons 206, the selection of the setting information selectedfrom the setting information view 204 is released.

In such a screen, the category icon, the setting information, and theoperation button 206 are selected using the cursor 220. In the presentembodiment, the cursor 220 is moved in response to a gesture of anoperation body of a user. At that time, a cursor movable region 210showing the movable region of the cursor 220 is set around the cursor220. The cursor movable region 210 of the present embodiment is acircular cursor as shown in FIG. 1, for example. The cursor movableregion 210 and the cursor 220 displayed in the cursor movable region 210are moved together in the display region 200 in response to apredetermined gesture of the operation body of the user.

In the operation control device according to the present embodiment, themovement of both the cursor movable region 210 and the cursor 220 aremoved together in the cursor movable region 210 and the movement of thecursor 220 in the cursor movable region 210 can be independentlyperformed. According to this constitution, the cursor movable region 210is pointed to an instructed object displayed in the display region 200and instructed by the cursor 220, whereby the cursor 220 can be easilymoved near the instructed object. The cursor 220 is finely moved in thecursor movable region 210, and consequently the cursor 220 can beaccurately pointed to the instructed object. Hereinafter, theconstitution of the operation control device, which can realize theintuitive and highly accurate operation of the cursor 220, and theoperation control method using the operation control device will bedescribed in detail.

<2. Constitution of Operation Control Device> [Hardware Configuration]

First, a hardware configuration of an operation control device 100according to the present embodiment will be described based on FIG. 2.FIG. 2 is a block diagram showing the hardware configuration of theoperation control device 100 according to the present embodiment.

As shown in FIG. 2, the operation control device 100 according to thepresent embodiment is provided with a CPU (Central Processing Unit) 101,a RAM (Random Access Memory) 102, and a nonvolatile memory 103. Theoperation control device 100 is further provided with a stereo camera104 and a display device 105.

The CPU 101 is operated as an arithmetic processing device and a controldevice and controls the entire operation in the operation control device100 in accordance with various programs. The CPU 101 may be amicroprocessor. The RAM 102 temporarily stores programs used in theexecution of the CPU 101 and parameters suitably changed in theexecution. They are connected to each other through a host busconstituted of a CPU bus. The nonvolatile memory 103 stores programs andarithmetic parameters used by the CPU 101. The nonvolatile memory 103can use a ROM (Read Only Memory) and a flash memory, for example.

The stereo camera 104 is an input device which can recognize a positionin a depth direction. In the stereo camera 104, two camerassimultaneously image a plurality of input portions, moved by a user,from different directions, whereby information in the depth direction ofthe imaged image can be recorded. Instead of the stereo camera 104, arange finder using parallax may be used, for example. The stereo camera104 is constituted of imaging means which images a gesture performed bya user and an input control circuit which outputs the imaged imageimaged by the imaging means to the CPU 101. The gesture of the user isobtained by the stereo camera 104, whereby various data can be input tothe operation control device 100, or a processing operation can beinstructed to the operation control device 100.

The display device 105 is an example of an output device which outputsinformation. As the display device 105, a CRT (Cathode Ray Tube) displaydevice, a liquid crystal display (LCD) device, an OLED (Organic LightEmitting Diode) device can be used, for example.

[Functional Configuration]

Next, a functional configuration of the operation control device 100according to the present embodiment will be described based on FIG. 3.FIG. 3 is a functional block diagram showing the functionalconfiguration of the operation control device 100 according to thepresent embodiment.

As shown in FIG. 3, the operation control device 100 according to thepresent embodiment is provided with an input information acquisitionpart 110, a detection part 120, a motion determination part 130, amovable region movement processing part 140, a cursor movementprocessing part 150, a display processing part 160, and a display part170.

The input information acquisition part 110 is an operation partobtaining position information of an object to be detected andcorresponds to the stereo camera 104 of FIG. 2. The input informationacquisition part 110 according to the present embodiment images a user,who performs operation control, to obtain the image. The image is usedfor determining the operation body moved by the user and obtaining themovement information of the operation body. The input informationacquisition part 110 outputs the obtained image to the detection part120.

The detection part 120 detects the operation body, moving the cursormovable region 210 or the cursor 220, from the image obtained by theinput information acquisition part 110 and obtains operation bodyinformation of the operation body. The detection part 120 recognizes theoperation body using the depth information of the image and an arbitraryimage recognition method. The operation bodies for use in the movementof the cursor movable region 210 and the cursor 220 are respectively setin advance. For example, a first operation body operating the cursormovable region 210 can be set to the left hand of a user, and a secondoperation body operating the cursor 220 can be set to the right hand.The detection part 120 recognizes the operation body from an image anddetects the position and shape of the operation body as the operationbody information to output the operation body information to the motiondetermination part 130.

The depth information of the operation body can be obtained using, forexample, a stereo image processing that can realize the measurement ofthe three-dimensional position of an object. In the detection of theoperation body, an image processing method such as a shape detection, askin color detection, and a texture matching method is used, andconsequently, the operation body and the position and shape thereof canbe estimated and detected from an image. The operation control deviceaccording to the present embodiment can be provided with a storage part(not shown) which stores information used in an operation controlprocessing. The storage part corresponds to the nonvolatile memory ofFIG. 2. The storage part can store the information of the firstoperation body operating the cursor movable region 210, the informationof the second operation body operating the cursor 220, and theinformation of the shape in a cursor movement mode that enables themovement of the cursor by each operation body.

The motion determination part 130 determines the movement (gesture) ofthe detected operation body based on the operation body informationreceived from the detection part 120. The motion determination part 130determines whether the detected operation body operates the cursormovable region 210 or the cursor 220. The motion determination part 130determines, based on the position and shape of the detected operationbody, whether or not the operation body is in a cursor movement mode inwhich the cursor movable region 210 or the cursor 220 is operated.

When the motion determination part 130 determines that the firstoperation body operating the cursor movable region 210 is in the cursormovement mode, the motion determination part 130 makes the movableregion movement processing part 140 to perform a processing for movingthe cursor movable region 210 in accordance with the movement amount ofthe first operation body. At that time, since the cursor 220 is moved inaccordance with the movement of the cursor movable region 210, themotion determination part 130 makes the cursor movement processing part150 to move the cursor 220 in accordance with the movement amount of thefirst operation body.

When the motion determination part 130 determines that the secondoperation body operating the cursor 220 is in the cursor movement mode,the motion determination part 130 makes the cursor movement processingpart 150 to perform a processing for moving the cursor 220 in accordancewith the movement amount of the second operation body. When the motiondetermination part 130 determines that the condition of the operationbody does not correspond to the above two conditions, the motiondetermination part 130 does not perform any output to the movable regionmovement processing part 140 and the cursor movement processing part150.

The movable region movement processing part 140 performs a processingfor moving the cursor movable region 210, displayed in the displayregion 200, in response to the movement of the operation body. Based ona current position information of the first operation body operating thecursor movable region 210, the movable region movement processing part140 calculates the direction and amount of movement of the firstoperation body from the position of the first operation body in aprevious time. The movable region movement processing part 140 thencalculates the direction and amount of the movement of the cursormovable region 210 in the display region 200 from the direction andamount of movement of the first operation body. The movable regionmovement processing part 140 outputs the calculated direction and amountof the movement of the cursor movable region 210 to the displayprocessing part 160. When the cursor 220 is located outside the cursormovable region 210 by the movement of the cursor 220, the movable regionmovement processing part 140 receives an instruction from the cursormovement processing part 150 to perform a processing for moving thecursor movable region 210 in accordance with the movement of the cursor220.

The cursor movement processing part 150 performs a processing for movingthe cursor 220 displayed in the display region 200 in response to themovement of the operation body. Based on the current positioninformation, the cursor movement processing part 150 calculates thedirection and amount of the movement of the first operation body,operating the cursor movable region 210, or the second operation body,operating the cursor 220, from the position of the first or secondoperation body in a previous time. The cursor movement processing part150 then calculates the direction and amount of the movement of thecursor 220 from the direction and amount of the movement of theoperation body. The cursor movement processing part 150 outputs thecalculated direction and amount of the movement of the cursor 220 to thedisplay processing part 160.

The cursor movement processing part 150 determines whether or not thecursor 220 is located outside the cursor movable region 210 by themovement of the cursor 220 by the second operation body. When the cursormovement processing part 150 determines that the cursor 220 is locatedoutside the cursor movable region 210, the cursor movement processingpart 150 outputs an instruction to the movable region movementprocessing part 140 to direct the movable region movement processingpart 140 to move the cursor movable region 210.

The display processing part 160 performs a display processing for movingand displaying the cursor movable region 210 and/or the cursor 220 basedon the information received from the movable region movement processingpart 140 or the cursor movement processing part 150. The displayprocessing part 160 outputs display information after the displayprocessing to the display part 170 to make the display part 170 todisplay the display information. The display part 170 corresponds to thedisplay device 105 of FIG. 2. As the display part 170, a CRT displaydevice, a liquid crystal display device, and an OLED device can be used,for example. Although the operation control device 100 of the presentembodiment is provided with the display part 170, the operation controldevice 100 may be used so as to connect to an external display device(not shown) without providing the display part 170.

The functional configuration of the operation control device 100according to the present embodiment has been described. By virtue of theuse of the operation control device 100, the cursor movable region 210and the cursor 220 can be independently operated, and therefore, thecursor 220 can be operated in a stepwise manner. According to thisconstitution, a user can operate the cursor 220 intuitively and highlyaccurately. Hereinafter, an operation control method using the operationcontrol device 100 will be described based on FIGS. 4 to 9. In FIGS. 7to 9, the same processing as the processing in the operation controlmethod described using FIG. 4 is assigned the same reference numeral.

<3. Operation Control Method Using Operation Control Device>

In order to operate the cursor 220 in a stepwise manner, the firstoperation body operating the cursor movable region 210 and the secondoperation body operating the cursor 220 are set in the presentembodiment. As those operation bodies, there can be used an operationbody which is easily operated by a user and is suitable for performing agesture that is easily associated with the operation of moving thecursor 220. As the first and second operation bodies, parts of the bodyor objects moved by a user such as a right hand and a left hand,different parts, both hands and one hand, and a sight line of a user andhis hand can be set, for example. Hereinafter, the operation controlmethod for the cursor 220 using the operation control device 100, whenvarious operation bodies are used, will be described.

[(1) Operation Control Using Right and Left Hands]

First, the first operation body operating the cursor movable region 210and the second operation body operating the cursor 220 are setrespectively to a left hand and a right hand, and the operation controlmethod for the cursor 220 using the operation control device 100 will bedescribed based on FIGS. 4 to 6. FIG. 4 is a flow chart showing anoperation control method using the operation control device according tothe present embodiment when right and left hands are used. FIG. 5 is anexplanatory view explaining the operation of moving the cursor movableregion 210. FIG. 6 is an explanatory view explaining the operation ofmoving the cursor 220 in the cursor movable region 210.

First, the operation control device 100 according to the presentembodiment images a user by means of the input information acquisitionpart 110 and detects the respective positions of the left hand operatingthe cursor movable region 210 and the right hand operating the cursor220 by means of the detection part 120 (step S110). The inputinformation acquisition part 110 of the present embodiment is the stereocamera 104, for example, and can obtain the position of the user and thedepth information from an imaged image obtained by imaging the user. Thedetection part 120 detects the depth information and the position of theleft and right hands, which are the operation bodies, from the imagedimage received from the input information acquisition part 110. Thedetection part 120 can recognize the left and right hands by using animage processing method such as a shape detection, a skin colordetection, and a texture matching method, using setting information suchas a human anatomy model that is obtained by modeling the positions ofregions and joints of the body previously stored in a storage part (notshown)s.

Subsequently, the motion determination part 130 determines whether ornot the left hand is in the cursor movement mode (step S120). As in thepresent embodiment, when an operational object is operated by a gestureof a user, setting is performed so that with regard to one operationbody, the motions of the operation body and the operation processing arerelated to each other in general. When the left hand is the operationbody, the movement processing of the cursor and a volume controlprocessing can be related to a gesture moving the left hand in left andright directions. At that time, in order to specify the processing,which will be performed, by the gesture currently being performed by theuser, a mode that is information showing the processing to be performedcan be set. For example, the mode can be set so that a state that theuser clasps the left hand shows the cursor movement mode in which themovement processing of the cursor is performed, and a state that theuser opens the left hand shows a volume control mode in which the volumecontrol processing is performed. By virtue of the provision of thecursor movement mode, the user sets the operation body to the cursormovement mode or releases the cursor movement mode of the operationbody, and thus the user can intentionally moves the cursor 220.

In step S120, whether or not the left hand operating the cursor movableregion 210 is in the cursor movement mode is determined. Namely, onlywhen the state of the left hand corresponds to the cursor movement modein which the cursor movable region 210 is moved and operated, the cursormovable region 210 and the cursor 220 are moved by the motion of theleft hand. An arbitrary gesture such as clasping a hand can be assignedto the cursor movement mode. In that case, when a user moves his claspedleft hand in the up, down, left and right directions within a space, thedetection part 120 determines that the left hand is clasped to therebydetermine whether or not the left hand is in the cursor movement mode.

When it is determined that the left hand is not in the cursor movementmode in step S120, the motion determination part 130 executes theprocessing of step S140. Meanwhile, when it is determined that the lefthand is in the cursor movement mode in step S120, the movable regionmovement processing part 140 and the cursor movement processing part 150move the cursor movable region 210 along with the cursor 220 inaccordance with the movement amount of the left hand (step S130).

The movable region movement processing part 140 performs processing ofroughly moving the cursor 220. More specifically, first, as shown inFIG. 5, unit regions 230 obtained by dividing the display region 200into a predetermined number of regions are set. In FIG. 5, one regionobtained by dividing the display region 200 into m1×n1 is set as theunit region 230. The movable region movement processing part 140 movesthe cursor movable region 210 and the cursor 220 in the up, down, leftand right directions in the unit of the unit region 230 in accordancewith the movement amount of the left hand LH. Thus, as the divisionnumber of the display region 200 becomes larger, the cursor movableregion 210 and the cursor 220 are more finely moved. Meanwhile, as thedivision number of the display region 200 becomes smaller, the cursormovable region 210 and the cursor 220 are more roughly moved.

For example, the display region 200 is divided into 32×32, and when theleft hand LH is moved by 1 cm, the cursor movable region 210 and thecursor 220 are moved in the moving direction of the left hand LH by oneunit region. According to this constitution, the movable region movementprocessing part 140 moves the cursor movable region 210 and the cursor220 by the movement amount corresponding to 1/32 of the display region200 at every movement of 1 cm of the left hand LH, whereby the cursor220 is roughly moved in the display region 200. The realization of therough movement of the cursor has advantages of free from the influencesof hand movement noise and an erroneous response to a slight movement ofthe left hand LH. In addition, the cursor 220 can be quickly moved nearthe vicinity of an operational object.

The motion determination part 130 determines whether or not the cursormovable region 210 is moved and then determines whether or not the righthand operating the cursor 220 is in the cursor movement mode (stepS140). As in step S120, only when the right hand is in the cursormovement mode in which the cursor 220 is moved and operated, the cursor220 is moved by the motion of the right hand. As in the above case, anarbitrary gesture such as clasping a hand can be assigned to the cursormovement mode of the operation body moving the cursor 220. When it isdetermined that the right hand is not in the cursor movement mode instep S140, the operation control device 100 does not move the cursor 220and terminates the processing. Meanwhile, when it is determined that theright hand is in the cursor movement mode in step S140, the cursormovement processing part 150 moves the cursor 220 within the cursormovable region 210 in accordance with the movement amount of the righthand (step S150).

The cursor movement processing part 150 performs processing of movingthe cursor 220 highly accurately. More specifically, first, as shown inFIG. 6, unit regions 245 obtained by dividing a predetermined region 240including the cursor movable region 210 into a predetermined number ofregions are set. In the present embodiment, although a quadrangle inwhich the circular cursor movable region 210 is inscribed is the region240, the present invention is not limited to the example, and the region240 and the cursor movable region 210 may be the same region. In FIG. 6,one region obtained by dividing the region 240 into m2×n2 is set as theunit region 245. The cursor movement processing part 150 moves thecursor 220 in the up, down, left and right directions in the unit of theunit region 245 in accordance with the movement amount of the right handRH.

For example, the region 240 is divided into 32×32, and when the righthand RH is moved by 1 cm, the cursor 220 is moved in the movingdirection of the right hand RH by only one unit region. According tothis constitution, the cursor movement processing part 150 can move thecursor 220 by the movement amount corresponding to 1/32 of the region240 at every movement of 1 cm of the right hand RH. Namely, comparedwith the case where the cursor movable region 210 and the cursor 220 aremoved by the left hand LH, the movement amount of the cursor 220 issmall when the hand is moved by the same distance, and the cursor 220can be finely moved in the display region 200. As the unit region 245 isreduced in size (for example, the unit region 245 is constituted of onepixel), the cursor 220 can be more finely moved by the right hand RH.

When the cursor 220 is moved in response to the movement of the righthand RH, the cursor movement processing part 150 determines whether ornot the cursor 220 is located outside the cursor movable region 210(step S160). The cursor movement processing part 150 calculates thedirection and amount of movement of the cursor 220 from the directionand amount of movement of the right hand RH to grasp the positionalrelation between the moved cursor 220 and the cursor movable region 210in the display region 220. The positional relation can be calculatedfrom, for example, coordinates of the moved cursor 220 in the displayregion 220 and a function showing the cursor movable region 210.

When it is determined that the moved cursor 220 is located in the cursormovable region 210 in step S160, only the cursor 220 is moved inresponse to the movement of the right hand RH, and thereafter, theoperation control device 100 terminates the processing. Meanwhile, it isdetermined that the moved cursor 220 is located outside the cursormovable region 210 in step S160, the cursor movement processing part 150instructs the movable region movement processing part 140 to move thecursor movable region 210 in response to the movement of the right handRH. According to this constitution, the cursor movable region 210 ismoved along with the cursor 220 (step S170). This processing can preventthe cursor 220 from being located outside the cursor movable region 210.

There has been described the operation control method using theoperation control device 100 according to the present embodiment whenthe right and left hands are used. In this operation control method, thecursor movable region 210 and the cursor 220 are roughly moved by theleft hand, and the cursor 220 is finely moved by the right hand. Sincethe cursor 220 can be moved in a stepwise manner, the cursor 220 can beintuitively and highly accurately operated.

[(2) Operation Control Using Different Parts]

Next, an example in which the cursor 220 is moved in a stepwise mannerby different joints will be described based on FIG. 7. FIG. 7 is a flowchart showing the operation control method using the operation controldevice 100 according to the present embodiment when different parts areused. In this example, the part from the elbow joint to the wrist is thefirst operation body operating the cursor movable region 210, and thepart from the wrist joint to the ends of the fingers is the secondoperation body operating the cursor 220. Namely, for example when thewrist is moved with the elbow joint as the fulcrum, the cursor movableregion 210 and the cursor 220 are moved in response to the movement ofthe wrist. When the palm or the lower parts such as fingers is movedwith the wrist joint as the fulcrum, only the cursor 220 is moved inresponse to the movement of the palm or the fingers.

In the above operation control method, first, the positions of thejoints of the body of the user are estimated from an image obtained bythe input information acquisition part 110, and the positions of theelbow joint and the wrist joint are detected by the detection part 120(step S110). In the detection of each joint, for example, as describedabove, the human anatomy model stored in a storage part (not shown) isused, and the joint positions are estimated from the positional relationbetween the respective parts of the body, whereby the joint positionscan be detected.

Subsequently, it is determined whether or not the hand moving andoperating the cursor movable region 210 and the cursor 220 is in thecursor movement mode (step S120 a). For example, when the right handoperates the cursor movable region 210 and the cursor 220, the cursormovable region 210 and the cursor 220 are moved only when the right handis in the cursor movement mode. For example, a state that the hand israised higher than a predetermined height (for example, the height ofthe waist) can be set as the cursor movement mode. According to thisconstitution, when the user raises the other hand which is not theoperation body operating the cursor movable region 210 and the cursor220, it is possible to prevent the cursor movable region 210 and thecursor 220 from being moved by mistake.

When it is determined that the right hand is not raised in step S120 a,the processing of step S150 a is performed. Meanwhile, when it isdetermined that the right hand is raised in step S120 a, the movableregion movement processing part 140 and the cursor movement processingpart 150 move the cursor movable region 210 and the cursor 220 inaccordance with the movement amount of the part located lower than theelbow joint (step S130 a). The part located lower than the elbow jointincludes the parts on the wrist side. For example, the arm includes theshoulder, the elbow, the wrist, the bases of the fingers, and the jointsand parts of the fingers. With regard to arbitrary parts of those parts,the parts located on the shoulder side are upper parts, and the partslocated on the finger side are lower parts.

In step S130 a, the movable region movement processing part 140 movesthe cursor movable region 210 and the cursor 220 together by themovement amount of the lower part, located lower than the elbow joint,in the moving direction of the lower part. The direction and amount ofmovement of the cursor movable region 210 and the cursor 220 in stepS130 a can be determined in a similar manner to step S130 of FIG. 4.Namely, the cursor movable region 210 and the cursor 220 are roughlymoved in the display region 200.

After that, the cursor movement processing part 150 moves only thecursor 220 in accordance with the movement amount of the part locatedlower than the wrist (step S150 a). Based on the detection resultobtained by the detection part 120, the cursor movement processing part150 moves the cursor 220 by the movement amount of the lower part,located lower than the wrist, such as fingers, in the moving directionof the lower part. The direction and amount of movement of the cursor220 in step S150 a can be determined in a similar manner to step S150 ofFIG. 4. Namely, the cursor 220 can be finely moved in the cursor movableregion 210.

When the cursor 220 is moved in response to the movement of the lowerpart located lower than the wrist, the cursor movement processing part150 determines whether or not the cursor 220 is located outside thecursor movable region 210 (step S160). When it is determined that thecursor 220 is located outside the cursor movable region 210 in stepS160, the movable region movement processing part 140 moves the cursormovable region 210 along with the cursor 220 in response to the movementof the lower part (step S170). Meanwhile, when it is determined that thecursor 220 is not located outside the cursor movable region 210 in stepS160, only the cursor 220 is moved in response to the movement of thelower part located lower than the wrist, and thereafter the operationcontrol device 100 terminates the processing.

There has been described the operation control method using theoperation control device 100 according to the present embodiment whenthe different parts are used. According to this operation controlmethod, the cursor 220 can be moved in a stepwise manner by moving thedifferent parts. Further, the operation control method of this examplehas advantage that the cursor 220 can be roughly or finely moved by thesame hand.

[(3) Operation Control Using Both Hands and One Hand]

Next, an example in which the cursor 220 is moved in a stepwise mannerby both hands and one hand will be described based on FIG. 8. FIG. 8 isa flow chart showing the operation control method using the operationcontrol device 100 according to the present embodiment when both handsand one hand are used. In this example, the both hands are used as thefirst operation body operating the cursor movable region 210, and eitherleft or right hand is used as the second operation body operating thecursor 220. Namely, when the both hands are moved in the cursor movementmode, the cursor movable region 210 and the cursor 220 can be moved.When only one hand which is set as the second operation body is moved,only the cursor 220 can be moved. Hereinafter, the hand as the secondoperation body is a primary hand.

In the operation control method, first, the detection part 120 estimatesthe positions of the parts of the body of the user from the image,obtained by the input information acquisition part 110, and detects thepositions of the both hands (step S110). Subsequently, it is determinedwhether or not the both hands detected in step S110 are in the cursormovement mode (step S120 b). As the cursor movement mode, a state thatthe right hand is clasped and the left hand is open can be set, forexample. The state of the hand can be recognized by using the imageprocessing method such as a shape detection, using setting informationsuch as the human anatomy model previously stored in a storage part (notshown).

The motion determination part 130 determines whether or not the bothhands are in the cursor movement mode from the detected hand state. Whenat least one hand is not in the cursor movement mode, the processing ofstep S140 b is performed. Meanwhile, when it is determined that the bothhands are in the cursor movement mode in step S120 b, the movable regionmovement processing part 140 and the cursor movement processing part 150move the cursor movable region 210 and the cursor 220 by the movementamount of the both hands in the moving direction of the both hands (stepS130 b). The direction and amount of movement of the both hands may bethe average of the directions and amounts of movement of the respectivehands or may be the direction and amount of movement of either one hand.The direction and amount of movement of the cursor movable region 210and the cursor 220 in step S130 b can be determined in a similar way tostep S130 of FIG. 4. Namely, the cursor movable region 210 and thecursor 220 are roughly moved in the display region 200.

Subsequently, the cursor movement processing part 150 determines whetheror not the primary hand is in the cursor movement mode (step S140 b). Atthat time, in the same manner as step S120 b, the cursor movement modeof the right hand as the primary hand may be a state that the hand isclasped, for example. In this example, in the case where the cursormovable region 210 and the cursor 220 are moved by both hands and thecase where only the cursor 220 is moved by the primary hand, it ispreferable that the cursor movement mode is set so that the both casescan be clearly distinguished. In this example, the cursor movement modesof the right hand as the primary hand are the same, and the case wherethe cursor movable region 210 and the cursor 220 are moved and the casewhere only the cursor 220 is moved are distinguished from the state ofthe left hand which is not the primary hand.

When it is determined that the primary hand is not in the cursormovement mode in step S140 b, the movable region movement processingpart 140 terminates the processing. Meanwhile, when it is determinedthat the primary hand is in the cursor movement mode in step S140 b,only the cursor 220 is moved in accordance with the movement amount ofthe primary hand (step S150). Based on the detection result obtained bythe detection part 120, the cursor movement processing part 150 movesthe cursor 220 by the movement amount of the right hand in the movingdirection of the right hand. The direction and amount of movement of thecursor 220 in step S150 can be determined in a similar manner to stepS150 of FIG. 4. Namely, the cursor 220 can be finely moved in the cursormovable region 210.

When the cursor 220 is moved in response to the movement of the righthand which is the primary hand, the cursor movement processing part 150determines whether or not the cursor 220 is located outside the cursormovable region 210 (step S160). When it is determined that the cursor220 is located outside the cursor movable region 210 in step S160, themovable region movement processing part 140 moves the cursor movableregion 210 along with the cursor 220 in response to the movement of theright hand (step S170). Meanwhile, when it is determined that the cursor220 is not located outside the cursor movable region 210 in step S160,only the cursor 220 is moved in response to the movement of the righthand, and thereafter the operation control device 100 terminates theprocessing.

There has been described the operation control method using theoperation control device 100 according to the present embodiment whenboth hands and one hand are used. According to this operation controlmethod, the cursor 220 can be moved in a stepwise manner by movingeither both hands or only one hand.

[(4) Operation Control Using User's Sight Line and his Hand]

Next, an example in which the cursor 220 is moved in a stepwise mannerby a sight line of a user and his hand will be described based on FIG.9. FIG. 9 is a flow chart showing the operation control method using theoperation control device 100 according to the present embodiment whenthe user's sight line and his hand are used. In this example, the cursormovable region 210 is operated by the user's sight line which is thefirst operation body, and the cursor 220 is operated by the user's handwhich is the second operation body. According to the operation controlmethod in this example, the sight line is directed in an approximatedirection in which the user wants to move the cursor 220, whereby thecursor movable region 210 and the cursor 220 can be quickly moved, andthe cursor 220 moved near an operational object can be operated manuallywith high accuracy.

In the above operation control method, first, the detection part 120estimates the positions of the joints of the body of the user from theimage, obtained by the input information acquisition part 110, anddetects the user's sight line and the position of the hand operating thecursor 220 (step S110). As described above, the position of the hand canbe detected by using the image processing method such as a shapedetection, using setting information such as the human anatomy modelpreviously stored in a storage part (not shown). The direction of theuser's sight line can be detected using the method described in JapanesePatent Application Laid-Open No. 8-308801, for example. Namely, twolight fluxes are projected onto the eyeballs of the user at differentincident angles, and the reflected lights of the projected light fluxesreflected from the eyeballs are received. Two image signalscorresponding to the reflected lights are generated, and a valuecorresponding to the position of the pupil of the eyeball is calculatedbased on the difference information of the image signals. The directionof the user's sight line can be detected based on the position of thepupil and a position of a Purkinje image of the eyeball detected fromthe reflected light from the eyeball.

Subsequently, it is determined whether or not the hand which is theoperation body and is detected in step S110 is in the cursor movementmode (step S120 c). In step S120 c, when it is determined whether or notthe cursor movable region 210 and the cursor 220 are moved in thedirection of the user's sight line, the cursor movement mode of theoperation body operating the cursor 220 is used. When the operation bodyoperating the cursor 220 is the right hand, a state that only the righthand is raised to not less than a predetermined height (for example, theheight of the waist) can be set as the cursor movement mode. In stepS120 c, the hand moving the cursor 220 is limited to one hand (the righthand in this example), whereby a false detection occurring when the bothhands are raised can be prevented.

When it is determined that the hand as the operation body is not in thecursor movement mode in step S120 c, the processing returns to stepS110, and the processing from step S110 is repeated. Meanwhile, when itis determined that the hand as the operation body is in the cursormovement mode in step S120 c, the movable region movement processingpart 140 and the cursor movement processing part 150 calculate thedirection of the user's sight line to move the cursor movable region 210and the cursor 220 (step S130 c).

At that time, for example, the cursor movable region 210 may be moved sothat the center of the cursor movable region 210 is located in the unitregion 230 of the display region 200 including the intersection betweenthe direction of the user's sight line and the display region 200. Thecursor 220 may be moved in synchronism with the movement of the cursormovable region 210 by the cursor movement processing part 150. Thecursor movable region 210 and the cursor 220 are moved in this manner,whereby the cursor movable region 210 and the cursor 220 can be roughlymoved in the display region 200.

Subsequently, when the hand as the operation body is moved in the cursormovement mode, the cursor movement processing part 150 moves only thecursor 220 by the movement amount of the hand in the moving direction ofthe hand (step S150 c). The direction and amount of movement of thecursor 220 in step S150 c can be determined in a similar manner to stepS150 of FIG. 4. According to this constitution, the cursor 220 can bemoved finely in the cursor movable region 210.

When the cursor 220 is moved in response to the movement of the handwhich is the operation body (the right hand), the cursor movementprocessing part 150 determines whether or not the cursor 220 is locatedoutside the cursor movable region 210 (step S160). When it is determinedthat the cursor 220 is located outside the cursor movable region 210 instep S160, the movable region movement processing part 140 moves thecursor movable region 210 along with the cursor 220 in response to themovement of the right hand (step S170). Meanwhile, when it is determinedthat the cursor 220 is not located outside the cursor movable region 210in step S160, only the cursor 220 is moved in response to the movementof the right hand, and thereafter the operation control device 100terminates the processing.

The operation control method using the user's sight line and his handhas been described. According to this operation control method, thecursor 220 can be moved in a stepwise manner by moving the user's sightline and his hand.

There has been described the operation control device 100 according tothe present embodiment and the operation control method using theoperation control device 100. According to the present embodiment, thecursor 220 can be roughly moved along with the cursor movable region210, and, in addition, only the cursor 220 can be finely moved. Thecursor 220 can be operated in a stepwise manner by a gesture, and thecursor 220 can be operated intuitively and highly accurately.

It should be understood by those skilled in the art that variousmodifications, combinations, sub-combinations and alterations may occurdepending on design requirements and other factors insofar as they arewithin the scope of the appended claims or the equivalents thereof.

For example, in the present embodiment, although the cursor movableregion 210 is a circular cursor, the present invention is not limited tothis example. For example, the cursor movable region 210 may be a cursorhaving a polygonal shape such as a quadrangle.

In the present embodiment, although the size of the cursor movableregion 210 is fixed, the present invention is not limited to thisexample. For example, the size of the cursor movable region 210 may bevaried, whereby the cursor movable region 210, having a proper sizecorresponding to the layout of a screen displayed in the display region200 in which the cursor 200 is moved, can be set by a user.

For example, when the size of an object instructed by the cursor 220 isslightly larger than the cursor movable region 210, the cursor 220 canbe easily moved near the object. Thereafter, a user moves the secondoperation body to fine adjust the position of the cursor 220 in thecursor movable region 210, and, thus, to move the cursor 220, wherebythe object can be accurately selected.

The size of the cursor movable region 210 can be changed by a gesture ofa user. For example, when the user faces his palms to separate the bothhands, the size of the cursor movable region 210 can be increased, andwhen the user faces his palms to put the both hands close together, thesize of the cursor movable region 210 can be reduced. At that time, asize change mode that enables the change of the size of the cursormovable region 210 is set, and the size of the cursor movable region 210may be changed only when a user intends to change the size.

The present application contains subject matter related to thatdisclosed in Japanese Priority Patent Application JP 2009-171053 filedin the Japan Patent Office on Jul. 22, 2009, the entire content of whichis hereby incorporated by reference.

1. An operation control device comprising: a motion detection part whichdetects an object to be detected, which is moved by motion of a user; amotion determination part which determines motion of the object to bedetected based on a detection result obtained by the motion detectionpart; a movable region movement processing part which moves a cursormovable region including a cursor operating an object displayed in adisplay region; and a cursor movement processing part which moves thecursor, wherein based on the motion of a first detected object detectedby the motion detection part, the movable region movement processingpart moves the cursor movable region along with the cursor in thedisplay region by a first movement unit, and based on the motion of asecond detected object detected by the motion detection part, the cursormovement processing part moves only the cursor in the cursor movableregion by a second movement unit smaller than the first movement unit.2. The operation control device according to claim 1, wherein when aregion formed by dividing the display region into a predetermined numberof regions is a first unit region, the cursor movable region includesone or two or more of the first unit regions, and the movable regionmovement processing part moves the cursor movable region along with thecursor in the unit of the first unit region in accordance with an amountof movement of the first detected object.
 3. The operation controldevice according to claim 1, wherein when a region formed by dividingthe cursor movable region into a predetermined number of regions is asecond unit region, the cursor movement processing part moves the cursorin the unit of the second unit region in accordance with an amount ofmovement of the second detected object.
 4. The operation control deviceaccording to claim 3, wherein when the cursor is located outside thecursor movable region when the cursor is moved by the second detectedobject, the movable region movement processing part moves the cursormovable region along with the cursor moved by the second detectedobject.
 5. The operation control device according to claim 1, whereinonly when the first detected object is in a movement mode that enablesmovement of the cursor movable region, the movable region movementprocessing part moves the cursor movable region, and only when thesecond detected object is in a movement mode that enables movement ofthe cursor, the cursor movement processing part moves the cursor.
 6. Theoperation control device according to claim 1, wherein the size of thecursor movable region is varied.
 7. An operation control methodcomprising the steps of: detecting an object to be detected, which ismoved by motion of a user; determining motion of the object to bedetected based on a detection result; moving a cursor movable region, inwhich a cursor operating an object displayed in a display region isdisplayed, and the cursor together in the entire display region by afirst movement unit based on motion of a first detected object; andmoving the cursor in the cursor movable region by a second movement unitsmaller than the first movement unit based on motion of a seconddetected object.